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EVALUATING PRE-PLANNED CURRICULUM OBJECTIVES 

INTRODUCTION 

If there is a single most Important influence which behavioristic psychology 
has had upon curriculum design and development, it is perhaps the requirement 
chat curriculum objectives be stated in terms of observable human behavior. 

The position taken by educational psychologists with a behavior is^tic or neo- 
behavioristic orientation is that one can determine the conditions most appro- 
priate for learning 6nly after the terminal behavior has been specified. Once 
terminal objectives are specified in performance terms, it is then possible to 
proceed to develop instructional sequences appropriate for promoting the 
acquisition of that behavior. 

Curriculum developers who accept the premise that objectives should be 
stated in terms of observable human performance argue that only in this way may 
achievement of objectives be determined and the effectiveness of a particular 
curriculum sequence be assessed (Taber, Glaser and Schaefer, 1965; Gagne"', 1965). 

Purpose ; 

l-Jhile many have argued the merits of specifying curriculum objectives in 
behavioral terms, the present study was not undertaken to support the contention 
that curriculum objectives be stated behaviorally. Rather, the present project 
was conducted for the following reasons: 

1. To develop a set of criteria for determining the extent to which 
curriculum objectives had been stated in terms of observable student 
behavior, and to specify a rationale for the criteria and technique. 

i 

2. To train a group of public school teachers to use the criterion 
instrument to analyze a stated objective of a pre-planned curriculum. 

3. To determine whether a person trained to evaluate curriculum objectives 
in terms of specified criteria could do so consistently. 
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4. To determine the extent to which the objectives of the mathematics 
continuum used in Individually Prescribed Instruction are stated In 
behavioral terms. 

BacUfiround : 

The fundamental principle guiding the activities of the project was that 
whatever technique would be adopted to determine the behaviorality of rtiated 
curriculum objectives, it should be simple enough to be reliably and effectively 
used by people who were neither curriculum nor subject matter specialists. 

A number of pilot studies were conducted to determine what approach would 
be most effective. These preliminary efforts revealed that typical pre-service 
and In-servlcc teachers tended not to think of educational outcomes in terms 
of human behavior. They tended, rather, to equate dispositional states with 
their behavioral concomitants. When presented with objectives varying in 
degree of behavioral specificity, and asked the question, "IThich of these 
objectives most clearly describes observable human behavior?", pilot subjects 
responded very nearly randomly or arbitrarily, and complained they did not 
understand. Even after attempts to clarify the question, subjects failed to 
reliably respond. These preliminary efforts confirmed our suspicions that any 
workable technique for determining the behaviorality of objectives would 
probably involve an analysis of behavioral objectives into their critical com- 
ponents . 

The prior analyses of instructional objectives by Mager (1962), Miller 
(1961), Gagne' (1965) and Taber, Glaser and Schaefer (1965), served as a proto- 
type for the present analysis. A review of these writings indicated that 
while there was some variability in the recommended number of components in a 
behavioral objective, there xjerc three components upon which all agreed. 
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1* A specification of the l;ind of beltavior x^jhich the learner is expected 
to perform; that is, an action word or verb. 

2. A description of the important conditions under which the behavior 
will be expected to occur; that is, a context (or sinnall in which 
the action takes place. 

3. A description of the criterion for the kind of behavior which is to 
be performed; that is, a statement of how well the learner must 
perform to have his behavior accepted. 

Some users of behavioral objectives have added a fourth component to the 
list, and while at least one well-known educational psycholosist (Gagne', 1965) 
has attempted to find commonality among these such a synthesis is not clearly 
feasible. Further, bringing in a fourth component, such as an object acted 
upon (Gagne), complicates an analysis since some degree of expertise in the 
language of a particular subject matter is required. This, of course, would 
have violated our guiding principle that the evaluation technique be useable 
by personnel relatively unsophisticated in the subject matter. 

Locus of Bohavioralitv . Inspection of the three components led to the 
immediate conclusion that behaviorality resides in the first component: “The 

kind of behavior which the learner is expected to perform." Attention must 
be focused on the character of the verbs contained in behavioral objectives 
in order to estimate behaviorality since it is the verb or action word of an 
objective vjhlch describes behavior. 

Curriculum developers advocating the use of behavioral objectives have 
been adamant that objectives contain a description of specific observable 
human behavior . However, an examination of the words recommended by these 
curriculum developers reveals that what is meant by behaviorality is 
"observability" (for example, "to write", "to recite", "to list", "to state", 
rather than "to know", "to understand", "to appreciate", Mager, 1902). A 
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clp$e look at many of the recomr.iended words shows \.hat oil terms are» in some 
•anse, equally identifiable as descriptions of behavior; however, not all can 
be said to denote directly observable behavior (for esiample, "to identify", 

"to differentiate", "to solve", "to compare", Mager, 1962; or, "to distinguish", 
"to demonstrate". Commission on Science Education, 1963)* When the behavioristic 
curriculum developer demands on answer to the question, **VJhat will the learner 
be doing when he is demonstrating that he has achieved the objective?", he 
seems to be saying to us, "What observable data are necessary to make the 
inference that learning has occurred?". 

* 

Another way of describing the problem of describing behavior consistently 
has been proposed by Handler and Kessen (1959). In a careful analysis of the 
language of science those authors state that the criterion for a term's 
acceptability x^ithin science is its invariant usage; that is, its consistent 
usage by all members of a particular language community. If a term is used by 
all members of the same linguistic community. invariantly, then, apparently, 
it is safe to assume that for that linguistic community the referent is clear. 

In other words, the presence or absense of a particular event (referent) may 
be inferred from the consistency x^ith which members of a particular language 
community use a term. Thus, when ten physicists use the same term "proton" 
at the same time, the inference to be made is that there is a consistent 

referent for that term. * 

\ 

It occurred to the present Investigators that whatever is present In the 
external x;orld of the curriculum developer is present regardless of x^hat terms 
ate used in their attempts to describe that world. VJhat is Important is that 
curriculum developers use the sane terms at the same time to apply to the same 
events so that agreement on outcomes of the instruction can be achieved. 

Apparently, the concern of those promoting objectives which include 

specific observable human behavior is that terms such as "know", "understand" 
o 

me 



and "appreciate*' are not used invarlantly by native speakers of the English 
language, while words such as "write", "identify" and "solve" are used 
invariantly. The presumed variance in usage of a term such as "know" would 
mean by implication that the stimulus occasion for using such a term is not 
invariant-that there is no event consistently labelable with the term "know". 

By equating behavioral! ty with observability curriculum developers have, 
apparently, attempted to achieve a high degree of invariance with respect to 
usage. In this way, equivocation over the successful attainment of objectives 

is to be eliminated. 

Prior to our analysis of the action word or verb in terms of invariance, 
it had seemed reasonable that certain behaviors could be categorized as occurring 
or- not occurring simply by observing them (e.g.. "writing or saying )» Other 
behaviors thought by some to be directly observable, such as to identify or 
"to classify", apparently were actually processes which were essentially as 
unobservable as "know" and "appreciate". Subsequent consideration led to the 
conclusion, however, that even actions such as "write" and "say" would sometimes 
not be identified invariantly, and that pursuit of the line of reasoning that 
one could tabulate the occurrence or non-occurrence of a behavior simply by 
watching the organism would lead to the same difficulties encountered by 
behavioristic psychologists attempting to define the bar pressing response. 
Skinner avoids this complication by attending not to the topography of the 
response but to the effect of the response upon the environment. Thus, the 
bar presiing response is defined in terms of an electrical contact made, 
regardless of the manner (or form of behavior) by which the depression of the 
bar occurred (e.g., pushing with snout, tall or forepaws). 

Following Skinner's lead, our next attempt was to develop a technique 
whereby our teacher- judges would categorize the action word of an objective 
in terms of its effect upon the environment. As with the earlier attempt to 
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r.t. the behaviorality of at, entire objective thia effort was remarkably un- 
aueceasful. A queation such as. "To what extent would you be able to obaerve 
the effect of the word faction word) ?" produced aecmingly random behavior and 
misunderstanding. Aonarently it would be necessary to describe the effect of 
the behavior aa well as the behavior itself in order for subjects to make a 

judgement of this type. 

In a behavioral objective the effect of a particular response is contained 
in the criterion, and our nrimary task here was to determine the behaviorality 
of the action words used. Since this aporoach tended to take us away from the 
behavior and focused our attention on the criterion instead, it too was abandoned 



Rstlonalc t 

The pilot work led us to develop a technique for evaluating curriculum 
objectives based upon the analysis of behavioral objectives into the three 
components upon which all writers agreed - action word, context and criterion. 

Action Word . Since it appeared to us that most task analysts and curriculum 
developers were equating behaviorality with observability in an effort to achieve 
invariance, our solution to the nroblem of determining behaviorality was to 
structure the task for our subjects as one of observation. The question, "To 
what extent could you directly observe the behavior described by the term 
(action word)?" was a question which seemed to produce meaningful and consistent 
responding by all pre-service and in-service teachers. Our assumntlon was that 
judgements of "direct observability" were an indication not only of the extent to 
which a term served to label observable behavior, but also (and oerhans more 
Importantly) the degree to which a term was invariantly used by a native speaker. 

Since we were primarily concerned with analyzing the behaviorality of 
instructional objectives of the type recommended by curriculum developers with 
Jc • behavioral bent it seemed appropriate to construct a scale within the limits 
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tttabllshed by the experta in this field. Specifically, the polar extremta of 
obaarvability have, eaaentially, been eatabliahed in the writinga of Miller, 

• 

Gagne', and Mager* Maximum obaarvability ia implied by worda auch aa "to under-- 
line” or "to point to”, vhilo maximum unobaervability is implied by worda auch 
as ”to Imow”, ”to understand” and "to appreciate”. 

Our prediction was aa follows: 

If the polar extremes of a continuum of observability are eatabliahed for 
a group of native speaKera of the English language, and those speakers are 
asked to locate a set of labels (action words) on that continuum, then, words 
such ns "write”, "say” and "underline” will tend to fall toward the easily 
observed” pole, while words such as "know”, "understand” and "appreciate will 
tend to fall toward the "difficultly observed” pole. Further, our prediction 
was that many of the words commonly used and recommended by curriculum developers 
in writing behavioral objectives (auch as "identify”, "solve" and "distinguish ) 
would fall at an intermediate position on the continuum . 

The class of words falling at intermediate positions would be particularly 
interesting since they could, in no sense, be thought of as directly observable, 
would probably be used less Invarlantly and finally would be subject to greater 
disagreements than terms rated as observable. Ml:h such a scale an attempt 
was made to determine the behavlorallty of instructional objectives. 

Context or Signal . Following specification of the behavior that the learner 
is expected to perform as port of an objective, the conditions under which that 
behavior is expected to occur must be described (Taber, Glaser and Schaefer). 

Or, as others have suggested, an indicator or signal which is to be the dis- 
criminative stimulus for the behavior (Miller) or, "words denoting the stimulus 
situation which initiates the performance" (GagnO oust be specified. 

As with the action verb component of a behavioral objective there is, or 
cD?r>an bo made, a differentiation among different levels of generality of the 






contextual or stimulus component. For example, the context "when given the 
folloi7ing set of addition problems: 2 + 2 * . 2 v 3 “ . 3 •!* 5 * 

and the direction 'write the sums of the above set of addition problems in the- 
blank spaces'" is a more specific context than "when given single digit addition 
problems". Clearly, the more general context of the latter implies that any 
of a class of problems may by given to the student for him to act upon 
(although it is not entirely clear whether those problems will be in the form 
of mathematical sentences or in columns with a plus sign beside each number 
and a line belotf). 

While the first more specific type of context is open to less interpreta- 
tion it does not suggest what inference should be drawn about the pupil's 
capability with respect to a general class of problems. Characteristic of the 
more specific context is the inclusion of the actual test question or test 
items as a part of the context . . The more general context would serve 

as a basis for generating a wide variety of problems which, presumably, the 
learner ought to be able to perform. 

The distinction between general and specific context is made here because 
it seemed to the present investigators that when curriculum developers writing 
behavioral objectives demanded " specific observable human behavior," oftentimes 
the specificity which is sought is determined not only by the action verb but 
also by the nature of the context involved. If the context is specific then 
the objective will appear to be more specific (and, perhaps to some, more be- 
havioral or observable since the objective has been described in greater detail. 

Criterion . The final component of a behavioral objective upon which there 
is general agreement is that of the criterion of acceptable performance . An 
examination of both the literature dealing with instructional objectives and 
available curriculum objectives revealed that there are generally two types 
of criteria employed by most curriculum writers. The first type of criterion 
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l8 that which is established by c::plici':ly stating the correct answer to a 
given objective or problem. For e::ample, in the objective “when given the 
problem: 5 + 2 ■ . the student writes 2 blank", the criterion for. 

an acceptable response is explicitly given as "7". Anyone attempting to score 
pupil achievement on the objective would be able to do so because the problem 
and it' i^ceptable answer "7" is explicitly stated. 

The second type of criterion frequently occurring is that which indicates 
some percentage or proportion of correct responding on the part of the student. 
This second type of criterion is exemplified in the objective "counts orally 
from one to ten at least six out of seven times correctly.'* That the student 
counts from one to ten once correctly may be unsatisfactory for some. To have 
his response accepted the student is required to perform correctly six or seven 
times, and at that time the criterion of acceptability can be applied. Another 
example of this type might be, "\;hen given a multiple choice examination on 
the nature of plastics the student will ansv;cr 907* of the questions correctly." 
Again a proportionality of total responding is specified as an accepted level 
of performance. 

It is easy to ignore the often made recommendation that a criterion for 
acceptability be explicitly stated by saying that a criterion is nearly always 
implied in a stated objective. Interestingly enough, ho\7ever, the criterion 
that is obviously implied often is not the criterion that is cmplpycd in an 
instructional situation.^. 



Although the objectives from the IPI math continuum seem to imply a criterion 
of 10D7* correct responses, users of this curriculum employ a criterion of 
C07* correct (Dover Public Schools). 
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Conclusion : 

The purpose of the present investigation was to determine whether a 
technique for analyzing behavioral objectives in terms of the critical components 
outlined above could be developed and reliably used by a typical group of in- 
service teachers. 

We believe that it is most important to note at this time that the above 
description of the components of behavioral objectives and the observation of 
behaviorality and specificity are made neither prescriptively nor evaluatively. 
Rather, an analysis of this type was undertaken to develop a technique for 
determining the “behaviorality*' of curriculum objectives which could be both 
and meaningfully used. In a sense, then, it is more appropriate to 
refer to this rationale and the subsequent procedures involved as an analysis 
rather than an evaluation . 

It might be reasonable to conclude, however, that should one wish to 
reliably communicate the objectives of a curriculum, the more specific the 
contc:;t, observable the behavior, and carefully stated the criterion, the more 
likely are people to agree that a desired objective has been achieved. On the 
other hand, as an objective becomes more specific and “hchavioral" it becomes 
less inclusive. This increased specificity, then, produces objectives which 
may seem superficial or educationally trivial. Thus, increased reliability 
of communication may be purchased at the risk of sacrificing linguistic 
sophistication and educational respectability. 
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PRQCEDURES 

Consistent with the decision to analyze the objective in terms of critical 
components, as described in the rationale, a set of instructional objectives 
were classified in terms of context, specificity, and criterion. However, the 
problem encountered in attempting to rate the behaviorality of an action word 
which is embedded within a completely stated objective (the "observableness" 
of the behavior named by the verb, is often confused with the specificity of 
the context), resulted in the decision to perform the analysis in two parts. 
First, the action words from IPI along with a larger set of words commonly 
found in educational objectives were rated on an observability dimension. 
Second, a sample of objectives from the IPI math continuum were classified 
in terms of the signal and criterion components. 

Subjects 

Thirteen in-service teachers and one principal participated at one time 
or another in the activity involved in the analysis of the instructional 
objective. Only 11 of those teachers and the principal were available during 
the entirety of the analysis. The subjects were teachers from both elementary 
and secondary schools r^ij^ing in grade level from first to twelfth. 

Determination of Observability 

The extent to which a verb labels an observable event was determined by 
having subjects judge or rate the event on a scale of observability. This 
scale was constructed with five points, ranging from presumably "easily 
observable" events such as to hit and toj^ite, -to "difficultly observable" 

events such as to believe and to sympathize. 

Since the purpose of the present investigation was to determine the 

behaviorality of the IPI objectives nearly all of the verbs contained in 
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worklng paper #20 of the Individually Prescribed Instruction Mathematics 
Curriculum (April 1| 1967) were extracted and included for rating. In all, 

45 different action words were taken from the IPI mathematic objective. 

As previously mentioned) a number of additional verbs often used in 
Instructional objectives were included in a set of verbs to be rated. These 
verbs (54) brought the total number of verbs rated to 99. 

The ninety-nine action words were typed in a single double-spaced column 
on four pages. In addition) a set of written instructions describing the 
nature of the task for the rater) a general description of the problem) and 
examples of the polar extremes of the rating scale served as a cover sheet to 
the four pages of verbs. The instructions along with the four pages of verbs 
served as a test booklet which was then distributed to each subject. The 
page order was randomized to reduce the effect of particular sequences upon 
tho ratings given by the subject. A sample test booklet Is Included as 
Appendix A of this report. In addition to the written Instructions on the 
cover sheet of the booklet the students were requested not to return to a page 
after it was once completed. 

The subjects performed their ratings at the same time in a group. Fifteen 
minutes wereallowed for completion of the rating. All subjects easily com- 
pleted the rating within the fifteen minute time period. 

Component Classification 

Classification of objectives on the basis of critical components was 
accomplished by providing the students with a prepared description of the 
components. This description of the components is Included as Appendix B 
to this report. The description of critical components was to serve as the 
only instruction given to the subject as to how the objectives were to be 
classified. After reading the description the students were required to 



classify a set of objectives randomly selected from individually prescribed 

instruction mathematics continuum. Subjects were provided with a classlfica* 

• « 

tion sheet in which to indicate their choice of category. 

Prior to the classification task it became apparent that many of the 
objectives from the IPI continuum were compound or complex in nature. That 
is, very often more than one operation or action is called for from the 
etudent, and more than one context is Included. In effect, what appears to 
be a aingle objective often turns out to be two or more objectives. Consequently, 
subjects involved in the analysis were directed to classify each objective 
only in terms of the first operation described. 

Subsequent to reading the instructions the subjects were given a randomly 
selected set of IPI objectives (approximately 45) and a classification check- 
list. The subjects were requested to classify in terms of presence or 
absence of the critical components of those objectives. However, subjects 
expressed considerable consternation with their task after having rated 
approximately 20 of the objectives. Tlie subjects stated that they were able 
to classify consistently at first, but as they proceeded they became confused. 
Consequently, rather than continue the subjects were requested to take home 
their descriptions for further study. In addition they were directed to 
generate one example of a general objective with context and criterion stated, 
and one example of the specific example with context and criterion stated. 

On the following day the objectives that the students had constructed 
were printed on a chalk board and a practice session ensued where subjects 
attempted to determine the presence or absence of critical components in one 
another s objectives. Approximately two hours were spent during this practice 



session. 
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One day later a different set of 41 IPI objectives was randomly selected 
and presented to the subjects for critical component analysis. An unlimited 
amount of time was slven for comnletion of the classification task» and all 
subjects completed the task within 45 minutes. 

The results of both critical component classification tests are included 
in the following section. 

RESULTS 

The results arc reported in terms of the two-part analysis regarding the 

£“■ 

observability of the verb in the component classification. 

Verb Observability 

The mean ratings and the variance of the ratings for the entire set of 
action words is contained in Table 1. The words are ranked from most observable 
to least observable. Appendix D contains the same action words ordered 
alphabetically with their respective mean ratings and variances. 



TABLE 1 



RANK-ORDER DISTRIBUTION OF MEANS FOR 99 VERB RATINGS 



TERMS 


MEANS 


VARIANCES 


to cover with a card 


1.00000 


0.00000 


to lever press 


1.00000 


0.00000 


to line-draw 


1.00000 


0 . 00000 


to mark 


1.00333 


0.03333 


to point to 


1.0G333 


0.08333 


to crossout 


1.16C66 


0.33333 


to underline 


1.16666 


0.33333 


to walk 


1.16666 


0.33333 


*to circle 


1.25000 


0.38636 


to repeat orally 


1.25000 


0.38636 


*to count orally 


1.25000 


0.75000 


*to say 


1.25000 


0.75000 


*to write 


1.33333 


0.42424 


*to put on 


1.41666 


0.44696 


*to read orally 


1.50000 


0.45454 
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*fta Sjhadc 
iTi li^er. 
t. to, name' 

* to, iilt in, 
ti label. ■ 

" ta state 
to. remove 
place 

tK tbil vh.at . 
Wdjaw: '■ 

'.tp^ identify in .writing 
’*'ttn^cSeck 
*.to" con 8 tract . 
match 
take. !away . 

* to" make 
%o ^arrange . 
tbiffhiah . 

'^’*^.toIreid " 

’^.tolplay . 

*tto relocate 
to. connect. 

*>.tolg£ve! 

’^-.to ^reject, . 

*tols(ile'ct 
tdxhVose 
*..to .partition. . 

*-,t 6 ^change . 

A . ^ ^ 

•.to ,usc 

'^*-to '.subtract 
*-td .perf orm' .. 

’^^•to^ .'total 
’^Jto iliyide 
order 

, * to. measure 

* to' add 
*to' supply 
*'to demonstrate 

. *toyrcgroup 
^ *to\ multiply 

* to" round ..off. . 

*to^ group 

.* tol complete . 

* to. respond . to . . 

* to' average 
to.' summarize . 
to^ inquire 

^ to." utilize 

* to. borrow . 1 
tolacl^owledge . • 



TABLE T (continued! 

MITANS- 

ll 5.0000: 
U55333: 
1)58333. 
1)66666: 
1)75000 
1 <75000 
l; 91666. 
1)91666 
Ii9l666. 

2 ;Q8533 : 

2 £16666 : 
2i25000- 
2 aSOQO • 

2 ma33 : 



2 .:4.16M 
2 /3m3 : 

2 .5a3JJi : 
2.38333:: 

2 .38333. 
2.36666 : 
2.166666 
2.36666 
2.250D0 ; 
2.25000 
2.83333. 
2.91666 
2.91666 

2.91666. . 
2.91666 
3.00000:. 

3.00000 
3.00333. 

3.0. 0333:. 
3.00333 
3.00333 
3.00333 
3.16666 
3.16666 . . 
3 . 16666 
3.16666 . 
3.23000 

3.250.00 . 
3.33333:.: 
3.33333 .: 
3.33333 
3.50000 
3.5D000 
3.58333 :: 
3.58333:.: 



■ VARIANCES 



0.61818 

0.44696 

0.81060 

0.96969 

1.11363 

1.47727 

0.62078 

0.99242 

1.17424 

0.99242 

1.42424 

1.29545 

1.29545 

0.78787 

1.15151 

0.99242 

0.62870 

0.62870 

0.81060 

1.71969 

0.60606 

1.15151 

1.33333 

1.11363 

1.47727 

0.51515 

0.44696 

0.99242 

1.17424 

1.35606 

1.81010 

1.81818 

O.C1060 

0.99242 

1.17424 

1.35606 

1.35606 

0.87870 

1.06060 

1.24242 

1.42424 

0.56810 

0.93101 

0.60606 

1.17424 

1.17424 

0.81818 

1.00000 

0.44696 

1.17424 
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TABLE 1 feontlnuod^ 



TERMS 



MEANS 



VARIANCES 



*to find 


3.66666 


. »Vto Identify 


3.Q3333 


to see 


3.G3333 


*to convert 


3.91666 


to distinguish 


4.16666 


*to solve 


4.25000 


♦to apply 


4.25000 


to develop 


4.33333 


♦to test 


4.33333 


♦to determine 


4.33333 


to generate 


4.33333 


♦to create 


4.33333 


to discriminate 


4.50000 


♦to recognize 


4.50333 


to discover 


4.75000 


to become competent 


4.75000 


to infer 


4.75000 


to like 


4.75000 


to analyze 


4.03333 


to be curious 


4.83333 


to conclude 


4.03333 


♦to deduce 


4.33333 


to feel 


4.83333 


to concentrate 


4.03333 


to perceive 


4.03333 


to think 


4.83333 


to think critically 


4.33333 


to learn 


4.03333 


to appreciate 


4.91666 


to be aware 


4.91666 


to know 


4.91666 


to wonder 


4.91666 


to realize fully 


5.00000 


to understand 


5.00000 



r ^Denotes IPI verbs. 



L.696% 

0.37370 

2.33333 

1.3S606 

0.87870 

0.93831 

1.11363 

0.42424 

0.42424 

0.60606 

0.78707 

1.15151 

0.63636 

0.44696 

0.20454 

0.38636 

0.38636 

0.38636 

0.15151 

0.15151 

0.15151 

0.15151 

0.15151 

0.33333 

0.33333 

0.33333 

0.33333 

0.38636 

0.08333 

0.08333 

0.08333 

0.03333 

0.00000 

0.00000 



Inspection of Table 1 shows that the verbs or action words extracted 



from IPI range in rate of observability from 1.3 (most observable) to 4.8 

(least observable) with a median rating of approximately 2.8. 

\ 

Since the variances are an indication of the consistency with which 
the various terms were rated, it is interesting to note that only 4 out of 
the 99 variances exceed 1.50. 



i 
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Comnonent Class ification 

The results of the first attempt at component classification are shown 
in Table 2. The left hand column denotes the level in the IPI continuum and 
the number of the IPI objectives under the level. The number in any box . 
denotes the number of people classifying that component of the objective in 
the category indicated by the column heading. The surface consistency of 
the data obviates the necessity for applying any statistical test. 



Table 2 

Frequency Distribution of Subject's Ratings 
on Selected Objectives (first test). 



Objective 
Number 



B- 1-Numeration 
B-2 
B-3 
B-4 
B-5 
B-6 
B-7 

E-l-Subtraction 
E-2 
E-3 

E-1 -Multiplication 
E-2 
E-3 
E-4 

D* 1-Measurement 
D-2 
D-3 
D-4 
D-5 

D-1 -Geometry 
D-2 
D-5 
D-6 



Soecif ic 


General 


Slenal i 


Criterion 






Stated 


Not Stated 


Stated 1 


Not Stated 


2 


0 


10 






10 




10 


10 






10 




10 




10 




10 




10 




10 




10 


1 


. 9 


1 


10 


1 


9 




10 


1 


9 




10 




10 


1 


9 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




10 




9 




9 




9 




9 




9 


1 


8 




9 




9 




9 




9 




9 


1 


8 




9 




9 


1 


8 


1 


8 




9 




9 




9 




9 




9 


2 


7 


1 


8 




9 




7 




8 


1 


8 


J 


\ 


1 




t 





The results of the component classification after the subjects have practiced 
is shown in Table 3. A total of 41 objectives from the IPI math continuum 



o 

ERIC 






wore presented in the second test. Again there was virtually unanimous agrte* 
ment on every classification of critical components for every one of the 



objectives. 



Table 3 



Frequency Distribution of Subject's Ratings 
on Selected Objectives (second test). 



Objective 

Humber 



I Specific .. General !■ 



C-l-Time j 

C-2 I 

C-1 Measurement | 
D-1 -Numeration 1 
D-2 
D-3 
D-4 
D-5 

D-l-Place Value j 
D-2 

D-2-Addition 
D-3 
D-4 
D-5 
D-6 
D-7 
D-C 

D* 1-Sub traction 
D-2 

E-1 -Money 
E-2 
E-3 
E-A 

E-1 -Time 
E-2 
E-3 
E-4 
E-5 

E-l-Measurement 
E-2 
E-3 
E-A 
E-5 
E-6 
E-7 

E-l-Gcometry 
E-2 
E-3 
E-4 
E-5 
E-6 



Signal I 

Stated I Not Stated ! 



12 

12 

11 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 



1 

1 



12 

12 

11 

12 

12 

12 

12 

12 

C 

12 

12 

12 

12 

12 

12 

12 

-.2 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

11 

12 

11 

12 

12 

11 

12 

12 

12 

12 

8 

12 

12 

12 



Criterion 

Stated Not Stated 



3 

1 

1 



1 

1 

1 

2 



( I 

« I 



1 

1 

1 



12 

9 

10 

11 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

12 

11 

12 

11 

12 

11 

12 

12 

11 

12 

12 

11 

11 

11 

10 

12 

11 

11 

11 



o 
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In addition to the data regardino the observability of the action verb 
and the critical component analysis for each of the objectives the judgement 
for the critical components for each of the subjects was tallied and compared 
with the group ratings. That data is included in Table 4. 



3ABLE 4 

HUMBBR OF DEVIATIONS FROM 

THE CROUP'S RATING ON SPECIFICITY, SIGNAL, AND CRITERION. 



Cliff Brown 
^om Comer 
Marjorie Devine 
’^Carol Freidhoffer 
«Bill Gibbs 

Jfohn Jaclsson • 

Barbara Kelly 
^Roberta Lewis-Lewis 
^^^racc Owen 
*Ardis Rassmus 
•ilancy Smith 
♦Edward Stephens 



Signal Criterion 



I 4 

* m 

1 
3 

- 4 

1 I 

3 I 

3 9 

1 I 




Several ratings although they deviated from the group rating were judged 
by experts (Deno and Jenkins) to be acceptable. The * denotes the number 

of tliaes a subject deviated from the group (yet was judged to be accept- 
able by an expert). 



Inspection of Table 4 enables a determination of the degree to which any 
subject's ratings were consistent with the group's ratings. Perhaps the best 
thing that one can say regarding the consistency or reliability of the various 
raters Is that only 1 of the 12 deviates markedly from the group. It is 
well to note that a deviation does not necessarily Indicate Incorrectness on 
the part of the rater. To determine whether or not a particular deviation 
was reasonable or acceptable, the Investigators Independently analyzed each of 
the objectives on which any one rater had deviated. As the table Indicates 

the Independent analyses of the judges confirm the possibility of particular 
o 

ERIC 
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deviations made by individual subjects. That is not to say that a deviation 

was judged as correct or incorrect, rather that there was enough ambiguity in 

of the objective to allow for alternative categorization or classifi- 
cation. 



DISCUSSION 

The results indicate that many widely used and recommended behavioral 
terms do not describe behavior which is as clearly observable as some have 
suggested. For example, the word “solve," which is frequently used to describe 
student behavior on various types of problems in mathematics, is only a little- 
more observable than the terms “to be curious," “to think critically," or 
for that matter "to appreciate." Words such as "identify," "average," 

"respond to,' "group," "regroup," “measure," (from the IPI mathematics 
continuum) are all at some intermediate position with respect to observability. 
One is left with the inescapable conclusion that it is inappropriate to 
suggest that these terms arc descriptive of overt human behavior. Rather, 
they might better be thought of as more invariant descriptions of behavior 
than the terms "know" and "appreciate," and less invariant descriptions than 
the terms "write" and "underline." 

Apparently words like "identify," "demonstrate,’* and "apply" are not 
rated as directly observable because (and this is a logical conclusion) there 
are a variety of ways in which one might identify or demonstrate or apply. 

For example, one might identify (which has a mean rating for observability 

of by^tirttng (1.0S) by ^saying CIVZ5I, ^tfiat is^^o by 

"circling" (1.25). 

There is, in fact, some evidence to indicate that the use of words like 
identify only obscures the observableness of the behavior. The action word 
"write," for example, was rated as quite observable (1.33), and the word 
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"Identify" has a rather high mean rating (3.83). To state in the objective 
that the student is "to identify in writing" is to actually produce an 
intermediate rating for observability (2.17). It is tempting to conclude 
that as any two action words are combined into a verb phrase observability 
is reduced by whatever degree the less observable term is rated as unobservable. 

Perhaps, as was suggested earlier, curriculum developers x^ith a 
behavioristic bent have become sensitive to the criticism of trivialness 
in the objectives which they have written. Subsequently, they may have 
fallen prey to using only "moderately observable" terms rather than seem 
unsophisticated in the subject matter, or in the use of the language. Rather 
than write objectives containing only the verbs "to underline," "to cross out," 
"to v;rite," "to say" they have opted for what might be called process words 
such as "identify," "solve," "recognize," "distinguish," "apply," "determine," 
and "average." At this point the question may be posed to the users of these 
terms: "What will someone be doing when he is identifying, recognizing, 

solving, or deducing?" Some curriculum developers (see for example Wallbesser 
and Gagne) have addressed themselves to this problem. Their solution has been 
to define "identify" and "distinguish" (not in themselves adjudged observable) 
in terms of verbs or action words which are given a high rating on observa- 
bility (in our case mean ratings close to 1.0). Such an approach would 
likely be a successful one as long as those definitions are always included 
with the set of objectives. Possibly a more parsimonious approach would be 
sioply^to usa^ the action terma itr ^Aosc tin finirt ions such as "write," ^circle," 
"underline." However, the former technique (i.e., defining in observable 
terms) has an advantage in that it suggests a more general or inclusive 
capability which the learner has acquired, and, consequently, avoids the 



criticism of trivialncss. 
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It Is worth noting that the arithmetic processes so frequently used as 
action words in IPI mathematics continuum received very consistent ratings 
oh observability (to add 3.0G| to divide 3.08» multiply 3.17, to subtract 2.92). 
The consistency of this rating is particularly remarkable in that each of 
these process words was presented on a different page, and, most likely, x^as 
not an artifact of the sequence or the procedures employed. Consistency among 
such processes which seem logically equivalent adds credibility to the scale 
of observability used for this analysis. 

Generally, the action words extracted from the IPI objectives are rated 
in the intermediate ranges of observability. Although the range of IPI verbs 
encompasses most of the potential range (from 1.3 to 4.3) the median rating 
given (2.8) indicates that almost as many verbs are rated toward the unobservable 
pole as toward the observable pole. The rankings given the verbs extracted 
from the IPI objectives approximate a normal distribution with the majority 
of the verbs falling in an intermediate range (2.0 - 3.9). (See Table 5.) 

TABLE 5 

FREQUENCY DISTRIBUTION OF RATIl^GS 
GIVEN IPI VERBS 

1.0 - 1.9 2.0 - 2.9 3.0 - 3.9 4.0 - 4.9 

12 19 19 7 

Ti«enty-six of the 57 verbs from the IPI objectives were given mean ratings 
of 3.0 or greater. The action verbs "to solve," "to apply," "to test," 

"to determine," "to create," "to recognize," and "to deduce," all from the 
IPI objectives, were given mean observability ratings greater than 4.0. 

A note might be made here of the utility of Appendix D (the alphabetical 
listing of the action v7ords and their rated means of observability). If, as 
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ve have assumed, the statement about the behavlorallty of any nartlcular 
objective is largely determined by the action uord for that objective, then 
the behavioralness of any particular IPX objective can be determined by 
extracting the action word from that objective and finding its mean rating for 
observability in the table. Further, if one wished to write objectives, 
■rather than analyze them in terms of observability, it would be possible to 
use the table as a basis for selecting words which might be expected to yield. 
Invariant usage in the natural language. 

Component Classification 

An inspection of the classification data leads to the conclusion that 
nearly every objective sampled from the IPX llathcmatics Continuum may be 
described as “a general behavioral objective with neither a signal nor 
criterion explicitly stated." The judges unanimously classified only 2 out 
of the 64 XPX objectives sampled as having a signal or context stated. None 
of the XPX objectives rated by the subjects contained an explicit criterion 
of acceptable performance as defined in the set of instructions. Similarly, 
no objective was classified as specific, by the investigator's criterion. 

The latter conclusion is not surprising since the pre-requisite condition 
for an objective to be classified as specific is that the signal or context 
must be explicitly stated or contained within the objective. As previously 
mentioned, only two of the objectives contained a signal, and in both cases 
the signal was a general stimulus situation rather than a specific problem 
to be responded to. 

Xt is Important to note that the results of this analysis with respect 




to the XPX objectives in no way should be construed as a criticism of the 
objectives. The task set by the authors was to establish an instrument 
for analysis within the context of available writings on the construction of 
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behavioral objectives. That the IPI objectives are not consistent with the 
form of instructional objectives which was induced from the writings does 
not necessarily make them unclear or poorly written objectives. 

A final statement should be made regarding the number of IPI objectives 
which were analyzed. For two reasons the investigators did not believe it 
to be necessary or possible to analyze all of the objectives from the IPI 
Mathematics Continuum. First, the consistency of the ratings and analysis 
given by the subjects indicated, as described earlier, that almost all of the 
IPI objectives were of a particular type with respect to specificity of 
context and the criterion. Second, the task of classifying the components 
of a behavioral objective becomes extremely tedious after about 35 objectives. 
Since little information is provided by additional classifications the 
decision was made that it would be unnecessary to require the subjects to 
analyze more than the number of objectives given. 
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APPENDIX A 



SAMPLE TEST BOOKLET 





This is a study daslgnad to datarmlnG th© extant to which various words 
are labels for behavior which is directly observable. For example, most would 
that the verb to hit labels behavior which you can see, while the verb 
"to believe" labels an internal state which cannot be directly observed. 

On the subsequent pages you will find a list of action words, or verbs. 
Your task is to rate each word on a scale from 1-5 from most observable to 
least observable as follows: 



Most Observable 

1 2 
to hit 
to bite 



3 



Least Observable 
4 5 

to believe 
to sympathize 



Words such as "to hit" and "to bite" are to be given a rating of 1, while 
words such as "to believe" and "to sympathize" are rated as 5. 

Many of the words may not in your judgement be rated 1 or 5, and these 
you are to rate as 2, 3, or 4 as you see fit. 

I Remember, the rating you give is determined by the extent to which you 

I judge it possible to observe the behavior. 

I 



PLEASE PLACE THE NUMBER WHICH YOU GIVE AS A RATING IN FRONT OF THE WORD 



to recognize 

to identify in writing 

to take away 

to finish 

to average 

to put on 

to repeat orally 

to solve 

to see 

to measure 

to find 

to analyze 

to mark 

to discriminate 
to read orally 
to place 
to make 
to infer 
to write 
to feel 
to complete 
to distinguish 
to subtract 
to state 
to play 
to give 
to be aware 
to supply 
to wonder 




to create 



to understand 



to concentrate 
to name 
to use 
to number 
to read 

to acknowledge 
to shade 
to discover 
to underline 
to add 
to test 

to realize fully 
to locate 
to select 
to utilize 
to think 
to convert 
to appreciate 
to regroup 
to borrow 
to draw 
to point to 
to construct 
to count orally 
to walk 
to be curious 
to respond to 
to label 



to connect 



to learn 



to choose 
to determine 
to inquire 
to like 
to partition 
to apply 
to remove 

to become competent 
to think critically 
to cover with a card 
to say 
to develop 
to arrange 
to draw 
to circle 
to divide 
to perceive 
to select 
to fill in 
to total 
to group 
to lever press 
to match 
to identify 
to tell what 
to reject 
to generate 
to deduce 



to conclude 



to summarize 



to know 

to demonstrate 
to check 
to order 
to round off 
to cross out 
to perform 
to change 
to multiply 



to line-draw 



APPENDIX B 



INSTRUCTIONS FOR COMPONENT CLASSIFICATION 



Several veil known educators (Hiller, Gagne", Mager) have agreed that well-stated 
behavioral objectives should include at least the following characteristics; 

1, A specification of the kind of behavior which the learner is 
expected to do, that is the action. 

2, A description of the important conditions under which the 
behavior will be expected to occur, that is the signal or 
context for the action to take place. 

3, Description of a criterion of how well the learner must perform 

to have his behavior considered acceptable, that is, some statements 
of a criterion of acceptability. 

For the moment, we will concern ourselves with only the 2nd and 3rd characteris- 
tics, the signal and the criterion for acceptability. 

First, let us determine if a given objective includes the statement of context 
(A) or if the writer of the objective has neglected this aspect (B). 

Below are examples of two objectives with 

CONTEXTS STATED 

A. 1. Given the command *'sav the numbers from 1 to_10_*^ counts orally 

from one to ten. 

2. Presented with pairs of two numbers connected by an addition sign (-f), 
writes the number that is the total of the pair. 

(signal underlined) 

Below are examples of two objectives with 

CONTEXTS NOT STATED 

B. 1. Demonstrates mastery of addition facts through sums to* 20. 

2. Solves addition and subtraction problems. 

A second related classification of behavioral objectives involves an objective's 
degree of specificity. An objective may be either general or specific depending 
on whether the test condition is explicitly included in the statement. Some 
examples will clarify this. 

SPECIFIC 



(test included) 

' . 1. Presented with the statement "regroup the following numbers 

(test included) 65 « 60 + « 50 + . 

160 ** 68 * 70", 



writes the numbers in the blanks that complete each operation. 



2 



2. When given the following problems 

9-4 

90-40 * _ 

(test included) 

900 - 400 * . 
9000 - 4000 * . 

writes the numbers indicating the difference; 

GENERAL 



(test not included) 



D. 1. When given several states of the union - and told to write their 
capitol cities ^ writes their capitols. 

2. VIhen presented with a series of names, writes the names in 
alphabetical order. 

An eKamination of examples A (1., 2.) and B (1., 2.) in terms of the specific 
general classification may be helpful. 

Look back to each of those examples now and try to determine if you would 
classify it as general or specific. 

According to the definition only example A - 1. is specific since none of the 
other three explicitly stite the test. 



By now. you may have discovered a rule of thumb to aid your classification. 
The rule states "a signal must be stated . explicitl y if the ° . 

classified as specific. If no signal or contex t is ““ted the objective is 
necessarily general. This cannot be interpreted to. mean that if a signal is 
present the objective is specific, only that for an objective to be specific 

a signal must be present. 



The final classification concerns the presence or absence of an explicitly 
stated criterion. For the purposes of our classification we will consider two 
kinds of criteria: (a) answer stated and (b) percentage of correct responses. 

The appropriate criterion statement is contingent on an objective s specificity 
or generality, that is, the criterion statement for a specific objective £i!£. 
correct a nswer , while the criterion statement for a general objective may 
either the correct answer or a percentage of correct responses. 



E. 1. When given the problem "5 ••• 2 ” ", writes 7. in the blank. 

2. When given the problem "5 + 2 = writes the correct answer in 

the blank. 

3. When given the problem "5 + 2 - writes the correct answer in 

the blank nine out of ten times. 
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Both objectives E-1, E-2, and E-3 are specific objectives since all contain the 
test conditions. Hox^ever, only E-1 contains an explicit statement of the 
criterion; that is, the answer 7. 



Consider three more onjectives. 

F. 1. Counts orally from one to ten. 



2. Counts orally from one to ten by saying "one, two, three, four. five, 
six, seven, eight, nine, ten." 



3. Counts orally from one to ten at least six out of seven times. 



Each of the above three objectives are general objectives in that the test 
not eplicitly stated. This is especially easy to determine since none of 
the three contain a signal which is a necessary requisite for a specific 
instructional objective. 



is 



Remember now that we said in a general objective the criterion may be stated 
as either the correct ansx^er or as a percentage of correct responses. Which 
of the above three objectives fit this rule? 



Obviously, F-1 does not fall within the category since neither the 
ansx^er is stated, nor is there a statement of percentage of correct 
Consequently, F-1 falls into the category of criterion not stated. 



correct 
responses . 



However, F?2 fits the rule because the correct answer is explicitly stated, 
bjcctivc F-3 also fits the rule because a statement of percentage of correct 

responses is contained. Consequently both F-2 and F-3 fall into the category 
of criterion stated. 



Consider two more examples. 

G. 1. M,en presented «ith five addition problems, writes the correct answer 
to four out of the five problems. 

2. When presented with five addition problems x^rites the correct answers 
to them* 



Both objectives are general since neither contain the test. However G-1 
contains a criterion statement while G-2 does not. * 

Now with the sheet marked CATEGORIES FOR OBJECTIVES you should be able to 
classify objectives on the basis of the following categories: 

1. Specific - General 

2. Signal: Stated - Not Stated 

3. Criterion: Stated - Not Stated 

For practice, rate the following objectives. 




•4 



Obj. 1 



Given a set of pictures name the animals contained in the picture. 



Now check your rating with the 
checked the following columns! 
not stated. 



correct answer? You should have 
general) signal-stated, criterion- 



Obj. 2 



Multiplication problems. 5 x 2 - 10. 
O X 12 72, and 2 x 100 « 2^, writes the underlined answers. ~ 



You should have checked the following columns: 
and criterion-stated. 



specific, signal-stated. 



beeln^«t?n.'’.r yo" hand for holi 

begin rating the objectives on the sheets handed to you. 



Otherwise 



o 
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APPENDIX C 



SAMPLE CL/iSSIFICATION SHEET 












Categories for Oblectives 



ojective 

jmber 


Specific 


General r 


Signal " tl 


Criterion 






Stated 


Not Stated 


Stated 


Not Stated 




























• 
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APPENDIX D 



THE 99 VERBS WITH MEANS AND VARIANCES 



% 



er|c 









PHENOMENA 



p 

PHENOMENA ‘ * 


X 


s' 


to acknowledge 


3.58333 


1.17424 


*rto add 


3.08333 


1.35606 


to analyze 


4.83333 


0.15151 


*to apply 


4.25000 


1.11363 


to appreciate 


4.91666 


0.08333 


*to arrange 


2.58333 


0.62878 


*to average 


3.41666 


0.81060 


to be av^are 


4.91666 


0.08333 


to become competent 


4.75000 


0.38636 


to be curious 


4.83333 


0.15151 


’^fto borrow 


3.58333 


0.44696 


*to change 


2.91666 


0.99242 


*to check 


2.25000 


1.29545 


to choose 


2.83333 


0.51515 


*to circle 


1.25000 


0.38636 


*to complete 


3.25000 


0.93181 


to concentrate 


4.83333 


0.33333 


to conclude 


4.83333 


0.15151 


*to connect 


2.66666 


1.15151 


*to construct 


2.25000 


1.29545 


*to convert 


3.91666 


1.35606 


*to count orally 


1.25000 


0.75000 


to cover with a card 


1.00000 


0.00000 


*to create 


4.33333 


1.15151 


to crosscut 


1 . 16666 


0.33333 


♦to deduce 


4.83333 


0.15151 


♦to demonstrate 


3.16666 


0.87878 


♦to determine 


4.33333 


0.60606 


develop 


4.33333 


0.42424 



i 




PHENOMENA 
to discover 
to discriminate 
to distinguish 
*to divide 
*to draw 
*to draw 
to fe&l 
*to fill in 
*to find 
to finish 
to generate 
*to give 
♦to group 
♦to identify 
♦to identify in writing 
to infer 
to inquire 
to know 
♦to label 
to learn 
to lever press 
to like 
to line-draw 
♦to locate 
♦to make 
to mark 
♦to match 
♦to measure 

^ *to multiply 

ERLC 



X 




s' 


A. 75000 




0.20454 


4.50000 




0.63636 


4.16666 




0.07378 


3.0C333 




0.G1060 


2.08333 




0.99242 


2.41666 




1.17424 


4.83333 




0.15151 


1.66666 




0.96969 


3.66666 




1.69696 


2 . 58333 




0.62878 


4.33333 




0.78787 


2.66666 




1.33333 


3.25000 




0.56818 


3.83333 




0.87378 


2.16666 




1.42424 


4.75000 


4 


0.33636 


3 . 50000 




0.81818 


4.91666 




0.08333 


1.75000 




1.11363 


4.75000 




0.38636 


1.00000 




0.00000 


4.75000 




0.38636 


1.00000 




0.00000 


2.66666 




0.60606 


2.41666 




0.99242 


1.0C333 




0.03333 


2.33333 




0.78787 


3.08333 




1.17424 


3.16666 




1.24242 



PHENOMENA 


X 


s* 


*to name 


1.5C333 


0.81060 


to number 


1.58333 


0.44696 


*to order 


3.0C333 


0.99242 


*to partition 


2.91666 


0.44696 


to perceive 


A. 83333 


0.33333 


*to perform 


3.00000 


1.81018 


*to place 


1.91666 


0.99242 


*to play 


2.50333 


1.71969 


to point to 


1.00333 


0.08333 


to put on 


1.A1666 


0.44696 


*to read 


2.58333 


0.81060 


*to read orally 


1.50000 


0.45454 


to realize fully 


5.00000 


0.00000 


V ^ 

*to recognize 


A. 50333 


0.44696 


*to regroup 


3.16666 


1.06060 


* to reject 


2.75000 


1.11363 


*to remove 

t 


1.91666 


0.62878 


to repeat orally 


1.25000 


0.38636 


*to round off 


3.16666 


1.42424 


*to respond to 


3.33333 


0.60606 


*to say 


1.25000 


0.75000 


to see 


3.C3333 


2.33333 


*to select 


2.75000 


1.47727 


^to shade 


1.50000 


0.81818 


*to solve 


A. 25000 


0.93181 


*to state 


1.75000 


1.47727 


*to subtract 


2.91666 


1.35606 


to summarize 


3.A1666 


1.17424 


:*to supply 


3,08333 


1.35606 






PHENOMENA 




X 


2 

8 


*to take away 




2.33333 


1.15151 


to tell what 




1.91666 


1.17424 


*to test 




4.33333 


0.42424. 


to think 




4.83333 


0.33333 


to think critically 




4.33333 


0.33333 


♦to total 




3.00000 


1.31818 


to underline 




1.16666 


0.33333.. 


to understand 




5.00000 


0.00000 


♦to use 




2.91666 


1 . 17424 


to utilize 




3.50000 


1.00000 


to walk 


•s 


1.16666 


0.33333 


to wonder 




4.91666 


0.08333 


♦to write 




1.33333 


0.42424 



o 



